Organization Validation for Service Inventory by JVickery-TBS · Pull Request #1668 · open-data/ckanext-canada

JVickery-TBS · 2026-04-09T20:13:31Z

We are able to consistently use the releases from the SI github to generate the CSV ref data we need. We have commands which load it into the tables which we define in an sql script in here. The part we were tripped on was relation between Service IDs and Program IDs, but there is no relation that is being defined or maintained currently so don't have to worry about that.

Just extended the temp datastore table to add an optional org_name to it, which we then use in the triggers. The Service ID one works fine, need to make the error message for it still.

The Program IDs one is gunna be interesting as Program IDs is a multiple select.

- Started working on reference tables for pd data.

- Continued script to generate service reference data. - Continued schema for ref tables. - Improved pd subcommand for loading ref data.

- Finalized script to generate service ref data. - Added more recombinant schema key/values.

- Made JSONB index. - Added choices_filter_query. - Added recombinant org_name to the temp ds table. - Started writing the database validation.

- Finalized ref value checks and errors in the db func. - Fixed up some other logic things.

- Started working on fiscal year function.

- Finalized fiscal year db function.

wardi · 2026-04-18T15:54:29Z

just those minor things otherwise LGTM

- Added change log file.

- Pyright fixes. - Flake8 fixes.

- Made service name fields computed ones. - Modified filter scripts for service name lookups.

- Filter script typo.

codecov · 2026-04-20T20:20:29Z

❌ 4 Tests Failed:

Tests completed	Failed	Passed	Skipped
309	4	305	0

View the top 3 failed test(s) by shortest run time

ckanext/canada/tests/test_service.py::TestService::test_required_fields

Stack Traces | 0.005s run time

self = <ckanext.canada.tests.test_service.TestService object at 0x7fe17af6b280>

    def test_required_fields(self):
        """
        Excluding required fields should raise an exception
        """
        chromo = get_chromo('service')
        record = chromo['examples']['record'].copy()
    
        expected_required_fields = ['service_description_en', 'service_description_fr',
                                    'service_type', 'service_recipient_type',
                                    'service_scope', 'client_target_groups',
                                    'program_id', 'client_feedback_channel',
                                    'automated_decision_system', 'service_fee',
                                    'os_account_registration', 'os_authentication',
                                    'os_application', 'os_decision', 'os_issuance',
                                    'os_issue_resolution_feedback', 'sin_usage',
                                    'cra_bn_identifier_usage', 'num_phone_enquiries',
                                    'num_applications_by_phone', 'num_website_visits',
                                    'num_applications_online', 'num_applications_in_person',
                                    'num_applications_by_mail', 'num_applications_by_email',
                                    'num_applications_by_fax', 'num_applications_by_other']
    
        for field in chromo['fields']:
            if field['datastore_id'] in chromo['datastore_primary_key']:
                continue
            if field.get('excel_required') or field.get('form_required'):
>               assert field['datastore_id'] in expected_required_fields
E               AssertionError: assert 'service_name_en' in ['service_description_en', 'service_description_fr', 'service_type', 'service_recipient_type', 'service_scope', 'client_target_groups', ...]

.../canada/tests/test_service.py:119: AssertionError

ckanext/canada/tests/test_service.py::TestStdService::test_required_fields

Stack Traces | 0.039s run time

self = <ckanext.canada.tests.test_service.TestStdService object at 0x7f1ae06c7040>

    def test_required_fields(self):
        """
        Excluding required fields should raise an exception
        """
        self._make_parent_record()
        chromo = get_chromo('service-std')
        record = chromo['examples']['record'].copy()
    
        expected_required_fields = ['service_standard_en', 'service_standard_fr',
                                    'type', 'channel', 'standards_targets_uri_en',
                                    'standards_targets_uri_fr']
    
        for field in chromo['fields']:
            if field['datastore_id'] in chromo['datastore_primary_key']:
                continue
            if field.get('excel_required') or field.get('form_required'):
>               assert field['datastore_id'] in expected_required_fields
E               AssertionError: assert 'service_name_en' in ['service_standard_en', 'service_standard_fr', 'type', 'channel', 'standards_targets_uri_en', 'standards_targets_uri_fr']

.../canada/tests/test_service.py:595: AssertionError

ckanext/canada/tests/test_service.py::TestStdService::test_max_chars

Stack Traces | 0.576s run time

self = <ckanext.canada.tests.test_service.TestStdService object at 0x7fef25c0d610>

    def test_max_chars(self):
        """
        Over max character field values should raise an exception
        """
        self._make_parent_record()
        chromo = get_chromo('service-std')
        record = chromo['examples']['record'].copy()
    
        expect_maxchar_fields = ['service_standard_en', 'service_standard_fr',
                                 'channel_comments_en', 'channel_comments_fr',
                                 'comments_en', 'comments_fr',
                                 'standards_targets_uri_en',
                                 'standards_targets_uri_fr',
                                 'performance_results_uri_en',
                                 'performance_results_uri_fr']
    
        for field in chromo['fields']:
            if field.get('max_chars'):
>               assert field['datastore_id'] in expect_maxchar_fields
E               AssertionError: assert 'service_name_en' in ['service_standard_en', 'service_standard_fr', 'channel_comments_en', 'channel_comments_fr', 'comments_en', 'comments_fr', ...]

.../canada/tests/test_service.py:767: AssertionError

ckanext/canada/tests/test_service.py::TestService::test_max_chars

Stack Traces | 0.958s run time

self = <ckanext.canada.tests.test_service.TestService object at 0x7f53f47ff7f0>

    def test_max_chars(self):
        """
        Over max character field values should raise an exception
        """
        chromo = get_chromo('service')
        record = chromo['examples']['record'].copy()
    
        expect_maxchar_fields = ['service_description_en', 'service_description_fr',
                                 'automated_decision_system_description_en',
                                 'automated_decision_system_description_fr',
                                 'os_comments_client_interaction_en',
                                 'os_comments_client_interaction_fr',
                                 'special_remarks_en', 'special_remarks_fr',
                                 'service_uri_en', 'service_uri_fr']
    
        for field in chromo['fields']:
            if field.get('max_chars'):
>               assert field['datastore_id'] in expect_maxchar_fields
E               AssertionError: assert 'service_name_en' in ['service_description_en', 'service_description_fr', 'automated_decision_system_description_en', 'automated_decision_system_description_fr', 'os_comments_client_interaction_en', 'os_comments_client_interaction_fr', ...]

.../canada/tests/test_service.py:406: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

- Created new workflow to semi-automate stuffs.

- Removed service name computed fields.

- Trigger type cast.

- Rebuild service inventory ref data. - Add fixme comment for overwritting dict assignment.

- Support multiple org names for one umd.

- Do not count new lines in max char func. - Added test coverage.

- Added functionality for the new recombinant CLI markers. - Added choice suffix to service inventory program_id. - Added test coverage. - Added support for old choice values in the recombinant webforms. - If `service_migration` marker is set, invalid program_ids will be suffixed with -INV.

- Renamed the datastore_user temp table to be more accurate as an app_context table.

- Better handle legacy choices. - Better handle choice suffixing. - Handle insecure firefox preventing window.confirm

- Added change log files.

- Pyright fixes. - Flake8 fixes.

- Various changes from feedback. - Change contextual datastore flags to just be one text[] db field.

- Improved suffix choices. - Minor logic changes for service inventory suffixes. - Fix newlines not displaying in pd datatables.

- Pytests don't like f string.

- Flake8 fixes. - Test fix for db func change. - Pyright fixes.

JVickery-TBS · 2026-08-04T20:15:46Z

@wardi I think this is ready for another review now.

wardi · 2026-08-16T23:07:47Z

otherwise LGTM

# Conflicts: # .github/workflows/pytest.yml # ckanext/canada/assets/public/canada_public.css ### RESOLVED.

- Bypass validation during migrations.

- Bring back service names.

- Fix truncating ref data tables. - Add command option to ignore file hashes.

- Revert.

- Form prefix code for service_id fields.

JVickery-TBS added 6 commits March 27, 2026 20:25

feat(dev): ref tables;

505d76e

- Started working on reference tables for pd data.

Merge branch 'master' into feature/ds-reference-tables

ec4635d

Merge branch 'master' into feature/ds-reference-tables

45fae29

feat(dev): ds ref tables cont;

2e72b9c

- Continued script to generate service reference data. - Continued schema for ref tables. - Improved pd subcommand for loading ref data.

feat(dev): ds ref tables cont;

10256b8

- Finalized script to generate service ref data. - Added more recombinant schema key/values.

feat(dev): ds ref tables cont;

fc740de

- Made JSONB index. - Added choices_filter_query. - Added recombinant org_name to the temp ds table. - Started writing the database validation.

JVickery-TBS requested a review from wardi April 9, 2026 20:13

JVickery-TBS assigned wardi and JVickery-TBS Apr 9, 2026

JVickery-TBS added enhancement Business Requirement labels Apr 9, 2026

JVickery-TBS mentioned this pull request Apr 9, 2026

Recombinant Org Based Triggers #1602

Closed

wardi reviewed Apr 10, 2026

View reviewed changes

Comment thread ckanext/canada/plugin/internal_plugin.py Outdated

JVickery-TBS added 3 commits April 16, 2026 19:40

feat(dev): ds ref tables cont;

433e216

- Finalized ref value checks and errors in the db func. - Fixed up some other logic things.

feat(dev): ds fiscal year func;

bcf369b

- Started working on fiscal year function.

feat(dev): ds fiscal year func;

65e40a8

- Finalized fiscal year db function.

wardi reviewed Apr 17, 2026

View reviewed changes

Comment thread bin/service_generate_reference_data.py Outdated

wardi reviewed Apr 17, 2026

View reviewed changes

Comment thread ckanext/canada/tables/service.yaml Outdated

wardi reviewed Apr 18, 2026

View reviewed changes

Comment thread ckanext/canada/tables/service.yaml Outdated

wardi reviewed Apr 18, 2026

View reviewed changes

Comment thread ckanext/canada/tables/service.yaml Outdated

JVickery-TBS added 5 commits April 20, 2026 15:23

feat(misc): changelog;

9af0578

- Added change log file.

fix(misc): syntax;

163fd73

- Pyright fixes. - Flake8 fixes.

feat(pd): service inventory;

eb89a03

- Made service name fields computed ones. - Modified filter scripts for service name lookups.

fix(pd): service inventory;

dfd01d8

- Filter script typo.

fix(pd): service inventory;

046e8f1

- Filter script typo.

JVickery-TBS added 3 commits April 21, 2026 13:32

feat(dev): github action;

1bd0da2

- Created new workflow to semi-automate stuffs.

fix(tests): service inventory;

6a9a1d1

- Removed service name computed fields.

fix(pd): service inventory;

9a82bb7

- Trigger type cast.

JVickery-TBS added 13 commits July 22, 2026 09:59

Merge branch 'master' into feature/ds-reference-tables

3eea3bb

Merge branch 'master' into feature/ds-reference-tables

4147a2a

feat(comments,build): script fix, ref data;

b2ce3a6

- Rebuild service inventory ref data. - Add fixme comment for overwritting dict assignment.

Merge branch 'master' into feature/ds-reference-tables

8549cdc

fix(bin): service ref data gen;

675ad7a

- Support multiple org names for one umd.

feat(db): max length new lines;

bf92612

- Do not count new lines in max char func. - Added test coverage.

Merge branch 'master' into feature/ds-reference-tables

ee8d7cc

refactor(db): datastore temp table;

adf0e53

- Renamed the datastore_user temp table to be more accurate as an app_context table.

feat(js): pd datatables;

c422648

- Better handle legacy choices. - Better handle choice suffixing. - Handle insecure firefox preventing window.confirm

feat(misc): changelog;

e4cf9f1

- Added change log files.

fix(lint): syntax;

cb720c5

- Pyright fixes. - Flake8 fixes.

fix(lint): syntax;

62e3cd3

- Pyright fixes. - Flake8 fixes.

wardi reviewed Jul 31, 2026

View reviewed changes

JVickery-TBS added 4 commits August 4, 2026 13:37

refactor(dev): feedback;

611d7cd

- Various changes from feedback. - Change contextual datastore flags to just be one text[] db field.

feat(pd): suffix choices;

bdc4ef4

- Improved suffix choices. - Minor logic changes for service inventory suffixes. - Fix newlines not displaying in pd datatables.

fix(tests): service filter;

19757bd

- Pytests don't like f string.

fix(syntax): lint;

31c160d

- Flake8 fixes. - Test fix for db func change. - Pyright fixes.

JVickery-TBS requested a review from wardi August 4, 2026 20:15

wardi reviewed Aug 16, 2026

View reviewed changes

Comment thread ckanext/canada/assets/datatables/pd_datatables.js

JVickery-TBS added 8 commits August 18, 2026 13:08

Merge branch 'master' into feature/ds-reference-tables

5ba54f3

# Conflicts: # .github/workflows/pytest.yml # ckanext/canada/assets/public/canada_public.css ### RESOLVED.

Merge branch 'master' into feature/ds-reference-tables

4dd7c21

Merge branch 'master' into feature/ds-reference-tables

aa7cbb3

feat(schema): logic;

9b6def8

- Bypass validation during migrations.

revert(pd): service names;

725831f

- Bring back service names.

fix(cli): load ref data;

ccd26b7

- Fix truncating ref data tables. - Add command option to ignore file hashes.

fix(tests): service;

2dd38ce

- Revert.

feat(pd): service;

1ea6eee

- Form prefix code for service_id fields.

Navigation Menu

Uh oh!

Organization Validation for Service Inventory - #1668

Organization Validation for Service Inventory#1668
JVickery-TBS wants to merge 58 commits into
masterfrom
feature/ds-reference-tables

JVickery-TBS commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wardi commented Apr 18, 2026

Uh oh!

codecov Bot commented Apr 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JVickery-TBS commented Aug 4, 2026

Uh oh!

Uh oh!

wardi commented Aug 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

JVickery-TBS commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wardi commented Apr 18, 2026

Uh oh!

codecov Bot commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 4 Tests Failed:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JVickery-TBS commented Aug 4, 2026

Uh oh!

Uh oh!

wardi commented Aug 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Apr 20, 2026 •

edited

Loading